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Abstract 

We consider the problem of approximating the probabihty mass of the 
set of timed paths under a continuous-time Markov chain (CTMC) that 
are accepted by a deterministic timed automaton (DTA). As opposed to 
several existing works on this topic, we consider DTA with multiple clocks. 
Our key contribution is an algorithm to approximate these probabilities 
using finite difference methods. An error bound is provided which indi- 
cates the approximation error. The stepping stones towards this result 
include rigorous proofs for the measurability of the set of accepted paths 
and the integral-equation system characterizing the acceptance probabil- 
ity, and a differential characterization for the acceptance probability. 

1 Introduction 

Continuous-time Markov chains (CTMCs) [T7] are one of the most prominent 
models for performance and dependability analysis of real-time stochastic sys- 
tems. They are the semantical backbones of Markovian queueing networks, 
stochastic Petri nets and calculi for system biology and so forth. The desired 
behaviour of these systems is specified by various measures such as reachability 
with time information, timed logics such as CSL[31 [5T], mean response time, 
throughput, expected frequency of errors, and so forth. 

Verification of continuous-time Markov chains has received much attention 
in recent years [3] . Many applicable results have been obtained on time-bounded 
reachability [3l [16] , CSL model checking |3l [21] , and so forth. In this paper, we 
focus on verifying CTMC against timed automata specification. In particular 
we consider approximating the probabilities of sets of CTMC-paths accepted by 
a deterministic timed automata (DTA) [T| [TT] . In general, DTA represents a 
wide class of linear real-time specifications. For example, we can describe time- 
bounded reachability probability "to reach target set G C S within time bound 
T while avoiding unsafe states U S" (G n t/ = 0) by the single-clock DTA 
Ai (Fig. [l]), and the property "to reach target set G C 5* within time bound 
Ti while successively remaining in unsafe states J7 C S* for at most T2 time" 
(G n [/ = 0) by the two-clock DTA A2 (Fig. [2]), both with initial configuration 
{qo,0). (We omit redundant locations that cannot reach the accepting state.) 
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The problem to verify CTMC against DTA specifications is first considered 
by Donatelli et al. |15j where they enriched CSL with an acceptance condition 
of one-clock DTA to obtain the logic CSL"""^. In their paper, they proved that 
CSL"^^ is at least as expressive as CSL and asCSL [31 [5], and is strictly more 
expressive than CSL. Moreover, they presented a model-checking algorithm for 
CSL"^^ using Markov regenerative processes. Chen et al. [13] systematically 
studied the DTA acceptance condition on CTMC-paths. More specifically, they 
proved that the set of CTMC-path accepted by a DTA is measurable and pro- 
posed a system of integral equations which characterizes the acceptance proba- 
bilities. Moreover, they demonstrated that the product of CTMC and DTA is a 
piecewise deterministic Markov process |14j , a dynamic system which integrates 
both discrete control and continuous evolution. Afterwards, Barbot et al. [5] 
put the approximation of DTA acceptance probabilities of CTMC-paths into 
practice, especially the algorithm on one-clock DTA which is first devised by 
Donatelli et al. |15| and then rearranged by Chen et al. |13 . Later on, Chen et 
al. |12j proposed approximation algorithms for time-bounded verification of sev- 
eral linear real-time specifications, where the restricted time-bounded case, in 
which the time guard x < T with a fresh clock x and a time bound T is enforced 
on each edge that leads to some final state of the DTA, is covered. Very recently, 
Mikeev et al. [TH] applies the notion of DTA acceptance condition on CTMC- 
paths to system biology. It is worth noting that Brazdil et al. also studied DTA 
specifications in [IV. However they focused on semi-Markov processes as the 
underlying continuous-time stochastic model and limit frequencies of locations 
(in the DTA) as the performance measures, rather than path-acceptance. 

Our contributions are as follows. We start by providing a rigorous proof 
for the measurability of CTMC paths accepted by a DTA, correcting the proof 
provided by Chen et al. [13] . We confirm the correctness of the integral equation 
system characterizing acceptance probabilities provided by Chen et al. |13j by 
providing a formal proof, and derive a differential characterization. This pro- 
vides the main basis for our algorithm to approximate acceptance probabilities 
using finite difference methods |20j . We provide tight error bounds for the ap- 
proximation algorithm. Whereas other works [HI HBl H] focus on single-clock 
DTA, our approximation scheme is applicable to any multi-clock DTA. To our 
knowledge, this is the first such approximation algorithm with error bounds. 
Barbot et al. [6] suggested an approximation scheme, but did not provide any 
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error bounds. 

The paper is organized as follows. In Section 2 we introduce some prelimi- 
naries. In Section 3 we prove the measurability of accepted paths, and prove the 
integral equations [13] that characterize the acceptance probability. In Section 
4 we develop several tools useful to our main result. In Section 5 we propose a 
differential characterization for the family of acceptance probability functions. 
Base on these results, we establish and solve our approximation scheme in Sec- 
tion 6 by using finite difference methods [501, which is the main result of the 
paper. Section 7 concludes the paper and discusses some possible future works. 

All integrals in this paper should be basically understood as Lebesgue Inte- 
gral. 

2 Preliminaries 

In this section we introduce continuous-time Markov chains jl7J and determin- 
istic timed automata [U [TTJ [T3| . 

2.1 Continuous-Time Markov Chains 

Definition 1. A continuous-time Markov chain (CTMC) is a tuple {S, L, P, A, C) 
where 

• S' is a finite set of states, and i is a finite set of labels] 

• P : 5' X iS I— >■ [0, 1] is a transition matrix such that X^ugs u) = 1 for 
aU s e S; 

• A : S* H- > M^Q is an exit-rate function, and C : S t-^ L is a, labelling function. 

Intuitively, the running behaviour of a CTMC is as follows. Suppose s is 
the current state of a CTMC. Firstly, the CTMC stays at s for t time units 
where the dwell-time t observes the negative exponential distribution with rate 
A(s). Then the CTMC changes its current state to some state u with probability 
P(s,m) and continues running from u, and so forth. The one-step probability 
of the transition from s to u whose dwell time lies in the interval / equals 
P(s, u) ■ J^^j A(s) • e"'''^*^* dt. Besides, the labelling function C assigns each state 
s SL label which indicates the set of atomic properties that hold at s. 

It is worth noting that under our definition, we restrict ourselves such that 
the rates of all states are positive. CTMCs which contain states with rate (i.e. 
deadlock states without outgoing transitions) can be adjusted to our case by (i) 
changing the rate of a deadlock state s to any positive value and (ii) setting 
P(s, s) = 1 and P(s, u) — for all u ^ s, i.e., by making a self-loop on s. 

Below we formally define a probability measure on sets of CTMC-paths. 

Definition 2. [3] Suppose M = {S,L,P,X,C) be a CTMC. An M-path tt 
is an infinite sequence sotoSiti . . . such that s„ G 5 and i„ G M>o for all 
n e Nq. In other words, the set of A^-paths, denoted by Path(A^), is essentially 
(S X IR>o)". Given an A^-path tt = sg^oSiii • . • , we denote s„ and i„ by 7r[n] 
and 7r(7i), respectively. 

A template is a finite sequence sqIq . . . s;_i/;_iSi such that I > 1, Sn & S 
for all < n < Z and /„ is an interval in ]R>o for all < n < Z — 1. Given a 
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template 6 = sq/o • ■ • si^ili^isi, we define the cylinder set Re as the following 
set: 

{tt e Path(A^) I 7r[n] = s„ for all < n < Z, and 7r(n) G /„ for all < n < 1-1} 

The probability space (Qs,J^s,Ss) over jM-paths with initial state s G 5 is 
defined as follows: 

• fis = {tt e Path(7W) I 7r[0] = s}; 

• J^s C 2^' is the smallest u-algebra generated by the following cylindrical 
family 

{Re I = sqIq ■ ■ ■ si-ih-isi is a template such that sq — s} 
of subsets of fig. 

• Bs '■ J^s ^ [0, 1] is the unique probability measure such that 

(Re) = Un=o {P(s«, s„+i) • A(s„) • e'^^^") * dt} 
for every template — SqIq . . . s/_i/;_iS;. 

Intuitively, the probability space {fig, J^s, Bs) is generated by all cylinder sets 
Re, where Bs{Re) is the product of one-step probabilities specified in 9. The 
uniqueness of is guaranteed by Caratheodory's Extension Theorem [5]. 

2.2 Deterministic Timed Automata 

Suppose A" be a finite set of clocks. A [clock) valuation over A" is a function 
77 : A" I— >■ M>o- We denote by Val(A') the set of valuations over X . Sometimes 
we will view a clock valuation as a vector with an implicit order on X . 

A guard (or clock constraints) over a finite set of clocks A" is a finite conjunc- 
tion of basic constraints of the form x n c, where xSA", nS {<,<,>,>} and 
c e No- We denote the set of guards over X by $(A). For each 77 € Val(A') and 
g G $(A'), the satisfaction relation \= g is defined by: ry |= a; ix c iff ri{x) n c, 
and 1= 01 A 32 iff H ffi ^'^'^ V H 92- Given g G <&(A'), we may also refer g 
to the set of valuations that satisfy g: this may happen in the context such as 
91 n g2, etc. Given X C X, r] G Val(A') and t € K>o, the valuations ri[X := 0], 
r] + t, and t] — t are defined as follows: 

1. a X £ X then t][X := 0]{x) := 0, otherwise r][X := 0]{x) := r]{x) ; 

2. [f] + t){x) := ri{x) -|- i for all a; G A" ; 

3. [rj t){x) :— rj{x) — t for all a; G A, provided that rj{x) > i for all a; G A' . 

Intuitively, rj[X := 0] is obtained by resetting all clocks of X to zero on 77, and 
77 -f- 1 resp. 77 — i is obtained by delaying resp. backtracking t time units from 77. 

Definition 3. [TJ 1131 111) A deterministic timed automaton (DTA) is a tuple 
(g,E,A, A,F) where 

• Q is a finite set of locations, and C Q is a set oi final locations; 

• S is a finite alphabet of signatures, and A" is a finite set of clocks; 

• A C Q X E X ^(A) X 2'^ X Q is a finite set of rules such that 

1. A is deterministic: whenever (gi, ai, gi, ATi, gj.), (92, 02, 52, ^^2, ^2) ^ 
A, if(gi,ai) = (92, 02) and giHga 7^ then {gi,Xi,q[) = (52 , -'^2 , ^2 ) ■ 

2. A is total: for all (q,a) G Q x I] and rj G Val(A'), there exists 
(g, a, X, g') G A such that 77 |= g. 
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Given qeQ^rjG Va\{X) and a G S, the triple {gl^, J e <^>{X) x2^xQ 

are determined such that (g, a, q^_a) G ^ is the unique rule satisfying 

V h g?.a- 

Definition 4. [H US] Let = (Q, S, X, A, F) be a DTA. A configuration of 
is a pair {q,rj), where q (z Q and 77 G Val(A'). A timed signature is a pair (a,t) 
where a G E and t G Ili>o- The one-step transition function 

: {Qx yal{X)) x (E x M>o) i-^ Q x Val(A') 

is defined by: K^i{q,v), = (q'^'^S l'? + i)[X^.+* := 0]) . 

We may represent K-^{{q,rj), (a,<)) = {q',ri') by the more intuitive notation 

-^^^ iq',v')"- We omit "yi" if the context is clear. 
Intuitively, the configuration K{{q,r]), {a,t)) is obtained as follows: firstly we 
delay t time-units at {q,r]) to obtain {q,T] + t); then we find the unique rule 
{q,a,g,X,q') G A such that rj + t |= 5; finally, we obtain K{{q,r]),{a,t)) by 
changing the location to q' and resetting rj + t with X. The determinism and 
the totality of A together ensures that /t is a function. 

Definition 5. ^ Let A ^ (Q, E, A", A, F) be a DTA. A timed word is an 
infinite sequence of timed signatures. The run of A on a timed word w — 
{{an, tn)}neKa with initial configuration (g, 77), denoted by Aq^riiw), is the unique 
infinite sequence {(^n, ??n)(an, in)}nGNo which satisfies that (qo:'7o) = (ItV) ^^'^ 

(g«+l,?7n+l) = K.^iiqn,r]n), (flnj^n)) for n > 0. 

A timed word w is accepted by A with initial configuration (g,??) {abbr. "i/j 
accepted by ^q,,,") iff ^^.^(li;) = {(g„,?7n)(a„,t„)}„gNo satisfies that q„ G F 
for some n > 0. Moreover, w is accepted by within k steps {k > 0) iff 

^g^^(u)) = {((Zn, '7n)(an, ^n)}neNo satisfics that qn G F for some < n < k. 

3 Measurability and The Integral Equations 

In this section, we provide a rigorous proof for the measurability of the set of 
CTMC-paths accepted by a DTA and the system of integral equations that 
characterizes the acceptance probability. The notion of acceptance follows the 
previous ones in [T5] . 

Below we fix a CTMC M = (5, L, P, A, £) and a DTA A = (Q, E, A", A, F) 
such that Yi — L. Given a finite or infinite word a, we denote by q;„ (n > 0) the 
rt-th signature, i.e., a — a^ai if a is infinite and a — a^ai . . . a^-i if 

a is finite with length k. Analogously, given a fc-dimensional vector t, we denote 

t = (to, . . . , tfc_i). 

Firstly, we formally define the notion of acceptance as follows. 

Definition 6. |13j The set of A^-paths accepted by A w.r.t s £ S, q E Q and 
7] G Va^A-), denoted by Path^'^^(s, q, r?), is defined by: 

Path^'^-^(s, g, ri) := {tt G Path(7W) | 7r[0] = s and is accepted by Aq^^} 

where Ct^ is the timed word defined by: (£7r)n — ('C(7r[7i]), 7r(n)) for all n > 0. 
Moreover, the set of Al-paths accepted by A w.r.t s, q and ry within k-steps 
{k > 0), denoted by Path^®"^(s, ry), is defined as the set of A^-paths tt such 
that 7r[0] = s and Cj^ is accepted by Aq.-q within k steps. 
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Note that specifies the behaviour of M. observable by an outside observer. 
By definition, we have Ufc>o Path^®-^(s, g, 77) = Path^®-^(s, g, 77). We omit 
"7W ® A' in "Path^®^" and "Path^^"^" if the underlying context is clear. 

Remark. We point out the main error in the measurability proof by Chen et 
al. [H]. The error appears on Page 11 under the label "(lb)" which handles the 
equality guards in timed transitions. In (lb), for an timed transition e emitted 
from q with guard x = K, four DTA Ae,Ae, Af,Af are defined w.r.t the 
original DTA A. Then it is argued that 

Paths'^iAe) = Paths'^ {Ae)\{Paths'^{A>) U Paths'^{A<)) 

This is incorrect. The left part Paths'^ (Ae) excludes all timed paths which 
involve both the guard x > K and the guard x < K (from q). However the 
right part does not. So the left and right part are not equal. □ 

Below we prove that Path(s, q, rj) is measurable under Qs for all (s, g, 77) € Sx 
Q X Val{X), and the integral-equation system that characterizes the acceptance 
probability [13 . We denote by (J) the characteristic function of a set J. We 

abbreviate (slcis)^^lc(s)'^q,c(s)) ^ (Sg, s> X^'.s: Qg, J- Given a rule 7 e A, 
we may denote 7 = (q(7), 0(7), 0(7), ^(7), q'(7)) • 

Note that 1Jj,>q Pathfc(s, g, 77) — Path(s, g, ?/). Thus in order to prove the 
measurability of Path(s, g, rj), it suffices to prove that each Pathfc(s, g, 7;) is mea- 
surable under Q^- To this end, we decompose Pathfc(s, g, 77) into subsets of paths, 
as follows. 

Definition 7. Suppose k eN. Define the set D'' C 5*^+^ x A*^ as follows. For 

all (a,/3) e 5'''+^ x A''', (a,/3) e D'' iff the following conditions hold: 

• a(/3„) = C{an) for all < ?i < fc — 1, and q'(/3„) = q(/3„+i) for all 
< 71 < A; - 2; 

• either q(/3„) G F for some < n < k — 1, or q'(/3fc_i) e F. 

Let D''{q,s) {{a, (3) G D'' \ (q(^o),ao) = ilys)}, for each g G Q and s e S. 

Definition 8. Suppose (a,/3) ^ {k > 1) and 77 G Va^A"). Define the 
set Path^^ C Path(Al) as follows. For all tt G Path(A^),7r G Path^ iff the 
following conditions hold: 

• 7r[7T,] = a„ for all < 77 < fc; 

• The run ^q(^„)_^(£^) = {(g„, 77„)(£(7r[7i]), 7r(77))}„>o satisfies that g„ = 
q(/3„) and rjn + 7r(7i) |= 0(/3„) for all < 77 < fc — 1. 

The intuition is that Path^ ^ is the set of A^-paths which visit the first fc -I- 1 
states in the state sequence a while A synchronizes with the timed path by 
taking fc rules from the sequence P (cf. Fig.|3]). From Definition [?) Definition [s] 
and the fact that A is deterministic, it is not hard to prove the following lemma. 

Lemma 1. For all k > 1, Pathfc(s, g, 77) = U{Path^^ | (a,/3) G D''{q,s)}. 
Furthermore, the union is disjoint, i.e., Path^ ^nPath^, ,^ = whenever (a, (3) ^ 
(a',/3'). 
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V / " V / /3" V ^"+1 / ' " V Vk~i J Pk-i \ Vk J 
Figure 3: The run given tt e Path^ 

Thus to prove that Pathfc(s, q, rj) is measurable, it suffices to prove that each 
Path^ ,j is measurable. To this end, we prove two technical lemmas as follows. 

Lemma 2. Let k > 1. For each /3 € A*^ and -q £ Val(A'), we define 

Jp^r, {t e M|o I ^[P^ V, t]« + t„ h QiPn) for all{)<n<k-l} 

where 77, t]„ G Val(A') (0 < 71 < fc — 1) is defined by: (of. Fig.^ 

''[P,V,'t]o ^ i^[/3,77,t]„+i = (j^[/3,?7,t]„ +t„)[X(/3„) := 0] . 

Then given any (a,/?) G D'^ and rj G Va^A"), tt G PathJ^ ^ i/f 7r[n] = a„ /or all 
<n < k and {tt{0), . . . , 7r(fc - 1)) G J/3,,,. 

Proof. Suppose tt G Path^,^. Let Aci(fjo).jj{C.^) = {(g„, 77„)(£(7r[n]), 7r(7i))}„>o 
and t := (7r(0), . . . ,7r(fc - 1)). By tt G Path^^, 7r[n] = for all < n < fc, 
and Qn — q(/3n) and ?7„ + 7r(n) 1= 0(/3„) for all < n < fc — 1. Then one 
can prove inductively on n that ?]„ = i^[/3,77,t]„ for all < n < fc — 1. Thus 
j^[/3, 77, t]„ + 1„ ^ QiPn) for all < 7i < fc — 1. It follows that t G Jp.rj- 

Suppose now that 7r[n] = a„ for all < 77 < fc and t := (7r(0), . . . , 7r(fc — 1)) G 
J^,,,. Denote A^(^i3g-)^jj{C^) = {(g„, 77„)(£(7r[77]), 7r(rt))}„>o. Since A is determin- 
istic, one can prove inductively on n that q„ = i(/3ti) and rjn — 7^[/3, 77, t]„ for all 
< 7^ < fc — 1. Then we have r]n + 7r(n) |= 0(/3n) for all < ti < fc — 1. □ 



V = 77, t]o 4^ • • • v[P, 77, t]„ ^ 7/[/3, 77, t]„+i . . . v[P, 77, t]fc_i 4^ ± 

Figure 4: The definition of i/[/3,77,t]„ 

Remark. One can prove inductively that for all < n < fc — 1 and for all a; G A": 

• I'lP, V, t]„(a;) + t„ = 77(2;) + X;r=o if 2; ^ U"=o^ ^(A); and 

• i^[/3,'7,t]„(x) +t„ = E"=m+iti if 3: G X(/3,„)\ (Ur=m+i -^(A)) for some 
unique m < n. 

Thus each j/[/3, 77, t]„(a;) + t„ is the summation of a possible constant and a 
consecutive segment of to, ... , t/j_i. □ 

Lemma 3. Let k>2. Suppose f3 e A'' and 77 G Val(A'). For all t G K>o, 

t e J;3,,, iffV + ^O 1= fl(/3o) fl'^rf t G >/,3,(r,+to)[X(/3o):=Ol ' 

where P = Pi . . . Pk-i and i = (ti, . . . ,tfe^i). 
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Proof. Suppose t G 
z/[/3,7y,t]„+i = 



^>Q. Note that for all < n < fc — 2 , we have 



/3, ry,t]i,t 



/3,(7y + to)[X(/3o) :=0],t 



Then we obtain: t g J^^jj 

iff i^[l3, rj, t]„ + t„ 1= 0(/3„) for all < n < fc - 1 

iff 77 + to h 0(/3o) and ry, t]„ + t„ |= fl(/3„) for all 1 < n < A; - 1 

iff r? + to h0(/3o) andz.[/3,(r; + to)[X(/3o) :=0],tl +t„hfl(/^n) 

for all < n < fc - 2 
iff ?7 + to h 0(/3o) and t e J^,(^+to)[3e(^)o):=o] ■ 



□ 



Now we prove the measurability result and the integral equations [13 . First 
we demonstrate that each closed subset of M>q is measurable when equipped 
with some a G 5''^+^. Below given a e S''^^ and W C M^p with fc > 1, we 
define Path [a, W] as the following set: 



{tt e Path(A^) I Tr[n] = a„ for all < 71 < fc and (7r(0), . . . , 7r(fc - 1)) e M^} . 

Lemma 4. Suppose a G 5*^+^ and W C M^p with fc > 1. If W is closed, 
then Path[a,T4^] is measurable under Qao- Furthermore, the probability mass of 
Path[a,W^] equals J^k D(q, t) • (W^)(t) dt, where 



k-l 



D(a, t) := H {P(a„, «„+i) • AK)e-^("")-*" } 



n=0 

Proof Let a G S''^^ and W C M^.^ closed with fc > 1. For every e > 0, define 
the hypercube set C 7'(M|J.q) as follows: 

HI := {[niQe, (mo + l)e] x • • • x [m^^ie, (mfe„i + l)e] | m„ G No for < n < fc-1} 

When equipped with a, each hypercube Iln^ol'^n^' (™" + ^)^] corresponds to 
the template ao[moe, (jno + l)e] . . . ak-i[mk-ie, (rnk-i + l)e]afe, which in turn 
corresponds to a cylinder set. Now define to be a hypercube-cover of W by: 



Further define Ck ■= Clnefi^k" where e„ = (|)"- We prove that W = Ck- 
It is clear that W C Ck- Suppose that Ck % W. Then there is a vector 
C G Ck\W. Since is a closed set, there exists a neighbourhood around ^ of 
diameter d in which all vectors are not in W. Then £^ ^ Ck since ^ ^ C| for all 
e < Contradiction. Thus W = Ck- Then it follows from n„ Path[a, C^"] = 
Path[a,Cfe] that Path[Q;, W] is measurable under 0^^. 

We have shown that W = HneN^fc"- Moreover, {C^"}„eN is monotonically 
decreasing since e„ = (i)". Thus (P^)(t) = hm (C^")(t) for all t G M|o- Note 
that 



/ D(a,t) • (/(SlfA) dt < / D(a,t)dt = [] P(a„,«„+i 



) < 00 
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Thenwehave lim /jj, ^ D(a, t) • (C^") (t) dt = /j^^^ D(a, t) • (W)(t) dt by Dom- 
inated Convergence Theorem. One can verify that J^k D(a,t) • (C^.")(t)dt 

equals the probabihty mass of Path [a, C^"]. Thus the probability mass of 
Path [a, W"] equals /jjfc^D(a,t) • (M^)(t)dt. □ 

We handle the measurability result and the system of integral equations 
simultaneously in the following theorem. Below we define V — S x Q x Val(A'). 

Theorem 1. For all (s, g,?]) € V and k >0, Path/j(s, 5, 77) is measurable under 
f2s- Furthermore, the family {probj, : V i— )• [0, l]}fc>o, where prob^(s, 77) is 
the probability mass of Pathfe(s, q, rj) under fls, satisfies the following properties: 
probo(s, g, 77) — {F){q); If q G F then probj,_|.]^(s, g, 77) = 1, otherwise 



Probfe+i(s,(7,7/) 




VP(,s,«) . prob, (u,q^+*, iv + m^+J := 0]) 



.ues 



dt 



Proof. First we prove that Pathfc(s, g, 77) is measurable. Let (s,g, 77) G V. The 
case fc = is easy: Patho(s, g, 77) is either or fig , depending on whether q ^ F 
or g e F. We prove the case when A: > 1. By Lemma [T] it suffices to prove that 
each PathJ^ ^ with {a, 13) € D^{q,s) is measurable. 

Let (a,/3) € D^{q,s). ByLemmapl PathJ^^ — Path[a, J^.^]. As is mentioned 



previously, J^_,, is specified by a finite conjunctive collection of linear constraints 

on {7r(77-)}o<n<fe-i: each takes the form Yl^n=ii ^ ^ where < < ^2 < 
k — 1, Ne {<, <, >, >} and c € M. We distinguish two cases below. 

Case 1: All n's present in the linear constraints are either < or >. Then 
Jp^r) is closed in M>o. Thus by Lemma |4j Path [a, J/s.r;] is measurable under 57s- 

Case 2: Some m is < or >. The point is that "<c = e | e > 0}" 

and ">c" likewise. Thus by the fact that is specified by a finite number 
of linear constraints, we have J^g^,, — UnsN"^^)) where J^^ is specified by the 
set of constraints obtained from J^_^ by replacing each occurrence of "<c" with 
"<c— (5) " and ">c" hkewise. Because each J^^ is closed, Path[a,J^',J is 
measurable under Vis- Then by Path[a, J^^,,] — {j^^^^PztU[a, J^^^], we obtain 
that Path^ ,^ is measurable under Og. 

Now we prove the integral-equation system for prob. Let {s,q,ri) e V. By 
definition, we have probo(s, g, 77) = {F){q) and probj,_|_j(s, g, 77) = 1 if g g F. 
We prove the relation between probj,_^]^ and prob^, when q ^ F. By Lemma [ij 
Pathfc_|_i(s, g, 77) is the disjoint union of {Path^ ,j | {a,f3) e £''^+^(g, s)}. Then 
probfc+i(s,g,7;) = Y.(a,f3)eD''+^q,s) P'^'^^^-n ' "^^^'''^ P^ob)^,,, is the probability 
mass of Path^ ^. We first prove that: 

prob^_^=/ D(a,t).(J^,,)(t)dt (t) 

given any m > 1 and (a, /?) € D™. Analogously, we distinguish two cases based 

on the types of constraints n that specify Jp^n- 

Case 1: All m's are either < or >. Then the result follows from Lemma |4) 
Case 2: Some N is < or >. We have shown that Jp^^ — Unew^^r;' 

where ^ is obtained from Jp^^i by relaxing < and > with (5)". Furthermore, 
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lim {Jg„){t) = (Jfl.„)(t) because {Jg „}n>i is monotonically increasing. By 

Leminaji the probability mass of Path [a, J^^] equals D(a, t) • (J^ j^)(t) dt. 

Thus by Dominated Convergence Theorem, we obtain (f). 

Consider prob^,, where fc > 1, (a,/?) € and q(/3o) ^ ^- Define 



d := ai . . . ak+i and ^ := /3i . . . 



By Lemmajaj (J/3,^)(t) = (0(/3o))(??+to)- (^/3.(r,+to)[X(/3o):=o]/ 
where t :— (ti, . . . ,tfe). Then by Fubini's Theorem and (f), we have 



. „,)(t) for allt G 



>o 



prob^ 



/ D(a,t)- (J^,^)(t)dt 

D(a,t) • (j0.(^+t)[3E(0„):=o])(t)dt dt 
D(aoai,t) • (0(^o))('7 + i) ' P''ob|(^+t)[3e(,9o):=o] 



R>o 



R>o 



where in the last step, we use the fact that G D'^. Below we prove the 

relation between prob^._|_i(s, 5, 77) and prob^ when q ^ F. If /c > 1, we have: 

Probfc+i(s,9,77) 



(a,0)eD'= + i(9,*) 



^ / D(aoai,i)-(0(/3o))(?? + O-prob|(^^j)[^(^^)^^o] 



dt 



EE E 

ue57eA,,s I (Q,;9)GD'=(q'(7),u) 



0(5-", t) • (0(7)) iv + t)- prob^ (^+t)[2e(^)^^o] dt 



= E E 



D(sw,t) • (fl(7))(77 + t) • I Pi'ob;^,(^+t)[x(7):=o] I dt 

Ua,/3)GD'=(q'(7),«) 



J2 J2 [ P(^^,")-AGs)e-^(^)*.(0(7)>(r] + t)-probfcKq'(7),(r? + t)[X(7):-O])dt 



MGS7GA, 



•^K>0 7eA,,, I 



(0(7)) + 



^ P(s, u) • prob,(ti, q'(7), (v + 01^(7) 0]) 



.ues 



+00 



A(s)e" 



^ P(s,w) • probfc (ii,q;'+*, (,7 + t)[X;'+* 0]) 



dt 



where Ag^s {7 G A | (q(7),a(7)) — (g,£(s))} and the last step is obtained 
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by the fact that the integrand functions are identical. If fc = 0, we have: 
probi(s,9,7y) 

{a,fS)eDHq,s) 
{a,l3)eD^{q,s) 

/•+00 

= / E {A(.)e-^(^)*-(0(7))(r/ + t)}dt 



76 Af.^ 



+ 00 



\{s)e 



-\(s)t 



J2 P(s, u) ■ probo {u, q^+*, (,y + i)[X^+* 0]) 



.ues 



dt 



where A^^ := {7 e A | (q(7),a(7)) = (g,>C(s)) and q'(7) e F} and the last 
equality is derived from the fact that the integrand functions are identical. □ 

The main result of this section is as follows. 

Corollary 1. For all (5,5,77) G V, Path{s, q,r]) is measurable under fig. Fur- 
thermore, the function prob : V [0, 1], for which prob(s, g, 7y) is the probability 
mass of Path(s, (7, ry) under fl^, satisfies the following system of integral equa- 
tions: If q € F then proh{s, q,r]) = 1, otherwise 



prob(s, q, 77) 



A(s)e 



-\(s)t 



^ P(s, u) ■ prob {u, q;'+*, (rj + t)[X2+* := 0]) 
lues 



dt 



Proof. It is clear that prob(s, g, 77) = 1 if g G F. Suppose q ^ F, then by 
Theorem [l] 



Probfe+i(s,(7,7y) = 
r+oo f 

Xis)e 



-\{s)t 



P(s, u) ■ prob, (t^, q^+*, (77 + ore* := 0]) 



.ueS 



dt 



Note that limfe_j.oo Pfobj, = prob. Thus by Monotone Convergence Theorem, we 
obtain the desired result by passing the lim operator into the integral. □ 



4 Equivalences, Lipschitz Continuity and The 
Product Region Graph 

In this section we prepare several tools to derive the differential characterization 
for the function prob. In detail, we review several equivalence relations on clock 
valuations [Tj and the product region graph between CTMC and DTA [T3] , and 
derive a Lipschitz Continuity of the function prob. 
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Below we fix a CTMC M = {S, L, P, A, C) and a DTA A = (Q, L, A, F). 
We denote by the largest number c that appears in some guard cc N c 
of A on clock X, by rj:^^^^^ the number maxa;^;^ T^, and by the value 

max{A(s) | s G 5}. We omit A4 or A if the context is clear. 

4.1 Equivalence Relations 

Definition 9. Two valuations rj, 77' e Val(A') are guard- equivalent, denoted 
by 77 =g 77', if they satisfy the following conditions: 

1. for all X eX, 7^{x) > iff i [x) > T^; 

2. for aU a; € A", if -qix) < and 7]'(x) < T^, then (i) int(77(a;)) = int(77'(a;)) 
and (ii) frac(?7(a;)) > iff frac(77'(x)) > 0. 

where int(),frac() are the integral and fractional part of a real number, respec- 
tively. Moreover, and 77' are equivalent, denoted by 77 ^ 77', if (i) 77 =g 77' and 
(ii) for all x,y ^ X, if r](x),ri'{x) < and rjly) , rj' (y) < Ty, then frac(77(x)) N 
frac(7/(7/)) iff frac(7/(a;)) ix ira.c{rf{y)) for all mg {<,=,>}. We will call equiv- 
alence classes of regions. Given a region [77]^, we say that [77]^ is marginal if 
vi^) ^ and frac(77(a;)) — for some clock x. 

In other words, equivalence classes of =g are captured by a boolean vector 
over X which indicates whether 77(x) > T^, an integer vector which indicates 
the integral parts on 77(0;) < and a boolean vector which indicates whether 
77(2;) is an integer when r]{x) < T^; equivalence classes of ^ is further captured 
by a linear order on the set {x e A" | 77(x) < T^} w.r.t frac(77(a;)). Below we 
state some basic properties of =g and ~. 

Lemma 5. /i/ The following properties on =g and ^ hold: 

1. Both =g and is an equivalence relation over clock valuations, and has 
finite index; 

2. if rj =g 77' then they satisfy the same set of guards that appear in A; 

3. If 7] ^ r( then 

• for all t > 0, there exists i' > such that + t ^ rj' + t' , and 

• for all t' > 0, there exists t > such that + t ^ ri' + t' . 

4. IfT]^ 77', then 77[X := 0] - t]'[X := 0] for all X C X. Moreover, for all 
77 e Val(A') and X C X, {r]'[X 0] | 77' e [77]^} is a region. 

Besides these two equivalence notions, we define another finer equivalence 
notion as follows. 

Definition 10. Two valuations 77, 77' £ Yb1{X) are bound-equivalent, denoted 
by 77 =b 77', if for all x £ X, either 77(2;) > and ri'(x) > T^, or 77(2:) — rj'{x). 

It is straightforward to verify that =b is an equivalence relation. The fol- 
lowing lemma specifies the relation between =b and prob, see Barbot et al. [6]. 
Below we present an alternative proof for integrity. 

Lemma 6. Let s E S , q E Q and rj, rj' E Val(A'). 7/77 =b v' j then prob(s, q, rj) = 
proh{s, q,rj'). 
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Proof. We prove that Path(s,q, 77) = Path(s, 77'). Suppose tt S Path (s, 77). 
Then the run = {(g„, 7y„)(£(7r[r7,]), 7r(n))}„>o satisfies that (jn £ F 

for some n > 0. Denote Aq,^i{CTr) = {{q'^,r]'^){C{-K[n]),'K{n))}n>o- We prove 
inductively on n that = and ?7„ =b 77,'i for all n > 0. This would imply 
that TT G Path(s, g, 77'). The inductive proof can be carried out by the fact 
that rjn =b implies ri„ + 7r(n) =b 7?^ + 77(71) and (ry^ + 7r(7i))[X := 0] =b 
(77; + 7r(7i))[X := 0] for all X (Z X . Thus Path(s, q, 7/) C Path (s, g, 77')- The 
other direction can be proved symmetrically. □ 

In the following, we further introduce a useful lemma. 

Lemma 7. For each rj e Val(A'), there exists ti > such that ri + t ^ i] + t' for 
all t,t' e (0,ti). For each rj £ Val(A') such that r]{x) > for all x £ X, there 
exists i2 > such that rj — t ^ rj — t' for all t, t' G (0, t2). 

Proof Define 7^' := {t]{x) - \ x e X a.nd t]{x) > T^}. If 77(2;) > for all 
clocks X, then we can choose ti to be any positive real number and t2 — minT?.'. 
Below we suppose that there is x € X such that 77(3;) < T^. 

Define TZ := {frac(?7(x)) \ x X and 77(0:) < Tj;}. Let Ci, C2 be the maximum 
and the minimum value of TZ, respectively. Note that < C2 < ci < 1. Then 
we can choose ti — 1 — ci. The choice of t2 subjects to the two cases below. 

1. C2 > 0. Then we can choose t2 — min{{c2} U TZ'}. 

2. C2 = 0. If 7?. = {02} then we can choose t2 = min{{l} U TZ'}. Otherwise, 
let c' > C2 be the second minimum value in TZ. Then we can choose 
t2 = min{{c'} U7^'}. 

It is straightforward to verify that ti , t2 satisfy the desired property. □ 

We denote 77+ to be a representative in {77 + i | i e (0,ti)}, and 77^ to be a 
representative in {77 — t | t £ (0,^2)}, where ti,t2 are specified in LemmajTj The 
choice among the representatives will be irrelevant because they are equivalent 
under ^. Note that if [77]^ is not marginal, then [7;]^ = [77+]^ = [77"]^. 

4.2 The Product Region Graph 

We define a qualitative variation of the product region graph proposed by 
Chen et al. |13j . mainly to derive a qualitative property of the function prob. 
The content of this subsection may be covered by the result by Brazdil et al. (lO] . 
Even though, we present it for the sake of integrity. 

Definition 11. The product region graph G^'^^ = [yM<^A ^^Mi&A^ 
and ^ is a directed graph defined as follows: V = S y. Q y. (Val(A')/ ^), 
((s, (7, r), (s', g', r')) € i5 iff (i) P(s, s') > and (ii) there exists r/ G r,i]' £ r' and 
t > such that [77 + i]^ is not a marginal region and {q' ,r]') ~ n{{q, 77), {C{s), t)). 
A vertex (s, q,r) gV is final ii q £ F. 

We will omit M®Ain G^®^ = lyM^A^ j^M^A-^ ^^le context is clear. 
The following lemma states the relationship between prob and the product region 
graph. Below we define 

7^,, := {0, 1} U {frac(77(x)) | a; e A" and 77(x) < T^} 

for each rj e Val(A'). 
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Lemma 8. For all (s,q,ri) G V, proh{s, q,ri) > iff {s,q, [rj]^) can reach some 
final vertex in G. 

Proof. It is clear that prob(s, g, 77) > iff prob f. {s, q,r]) > for some 

k £ Nq. We prove by induction on k that for all (s, q, 77) S V, if prob;.(s, g, 77) > 
then {s,q, [f]]^) can reach some final vertex in G. The base step A; = is easy. 
Suppose probj,_|_]^(s, 77) > with q ^ F. By Theorem[lJ we can deduce that 



+00 



A(s)e-^(^)*. ^P(s,u).prob,(^.,q^+*,('7 + i)re*:=0]) |> dt > 
.ues J J 

(1) 

Consider the regions traversed by 77 + f when t goes from to +00. Denote 
TZri = {wo,...,Wm} such that m > I and Wi > Wi+i for all < i < m. 
Note that wq = 1 and Wm = 0. We divide [0, +00) into open integer intervals 
(0, 1), (1, 2), (Ttnax - 1, T^ax), (Tmax, oo). For each n < Tmax, we further divide 
the interval (n, 77 + 1) into the following open sub-intervals, excluding a finite 
number of isolating points: 

(71 + I — 7i;o," + l — Wi),(72 + 1 — 7i;i,71 + l — W2), . . . ,{n + l — W,n-i,n+ 1 — Wm) 

Then we define the cluster 

I := {(77 + 1 - Wi,n + 1 - Wi+i) I < 7i < r,nax,0 < 7 < 777} U {(r,nax,+Oo)} . 

One can verify that for all / G X and t' ,t E I, rj + t ^ rj + t' . In other words, 
[ri + t]^ does not change when t is restricted to one of the intervals from I. 
By ^ , there exists / € I such that 



X{s)e 



-X{s}t ^ 



E 

.ues 



P{s,u) ■ prob, (u,q;'+*, (77 + t)[X;'+* 0]) 



dt>0 



This means that there is 7i* G S* and t* E I such that 

P{s,u*)-proh, {u*,q^+/,i7j + n[X^+/ :=0]) >0 

Since / is nonempty, [77 + t*]^ is not a marginal region. Thus there is an edge 
from (s,g, [77]^) to (w*, q^+**, [(77 + t*)[K'^+/ := 0]]^) in G. By the induction 
hypothesis, the vertex (7i*,q^+*', [(77 + t*)[X^+* := 0]]^) can reach some final 
vertex G. Then (s,q, [77] ~) can reach some final vertex in G. 

"<;=": Suppose (s, q, [77]^) can reach some final vertex in G. Let the path be 



(s, q, [7/]^) = {sk,qk-rk) (sfc-i, ^fc-i, ■ 



(so,9o,?'o) 



with go G F. We prove inductively on 77 < fc that prob„(s„, q„, 7;') > for all 
77' e r„. The case 77 = is clear. Suppose prob„(s„, g„, 7;') > for all rj' e r„. 
Let 77" e r„+i be an arbitrary valuation. By (s„+i, g„+i, r„+i) (s„,g„,r„), 
P(s„+i,s„) > and there exists rjn+i G r.n+i, rjn G r„ and t > such that 
[Vn+i + is not marginal and (g„,77„) = k(((7„+i, 7/„+i), (/:(s„+i), t)). By 
^ ^yn+i, there exists <' > such that 77" +t'^ Vn+i + 1, which implies that 



(77" + t')[x;'X 



0]^(77„+i+t)[X;':+^\t!+, :=0] (=77„) 
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By the fact that [rj" + t']r^ is not marginal, there exists an interval / C M>g with 
positive length such that for all t £ I, rj" + t ^ rj" + t' and 

irj" + r)[X;':+;,,^^^ := 0] ^ (ry" + t' Wl'^^^^, 0] - Vn ■ 

Thus by induction hypothesis, prob„ |^s„, g„, (ry" + t)[X^^+^^5^^^ := 0]^ > for 
all r e /. Hence 

{A(s)e-^(^)^ • [P(.s„+i, s„) • prob„ (.s„, 9„, (v" + r)[X^£+:,„^, := 0])] } dr > 
It follows that prob„^;^(s„+i, (7„+i, r/") > 0. □ 
4.3 Lipschitz Continuity 

Below we prove the Lipschitz Continuity. More specifically, we prove that all 
functions that satisfies a boundness condition related to =b and the system 
of integral equations specified in Corollary [T] are Lipschitz continuous. The 
Lipschitz continuity will be fundamental to our differential characterization and 
the error bound of our approximation result. 

Theorem 2. Let ft, : V i— > [0, 1] be a function which satisfies the following 
conditions for all s £ S , q £ Q and -q, rj' G Val(A'); 

• if I] =b v' then h{s, q, ry) = h{s, q, rj'); 

• if q (z F then h{s,q,f]) = 1, otherwise h{s,q,r]) is equal to the integral 



+ 00 



A(s)e- 



-A(s)t ^ 



J2 P(*, ") ■ h {u, q;'+*, {v + i)[x;'+* 0]) 



dt 



k lues 
Then for all s £ S, q <E Q and rj,ri' G Val(A'), if ^ if Woo ^ ^ then 

\h{s,q,7j) - h{s,q,ri')\ < Mi ■ - 7j'\\^ , 
where Mi := \X\ ■ AmaxT^ax • e^-"-^"- . 

Proof, li q G F, then the result follows from h{s,q,ri) = h{s,q,ri') = 1. From 
now on we suppose that q ^ F. To prove the theorem, it suffices to prove that 

\h{s,q,T]) -h{s,q,r]')\ < Amax^max • e^"""'^""" • 

if ll?? — < 1 ^^'i V'V' differ only on one clock, i.e. \{x G X \ r}{x) ^ 
ri'{x)}\ = 1. To this end we define (5(e) for each e G (0, 1) as follows: 

(5(e) sup {|/i(s, g, 77) - ft(s, g, 77')! | (s, g) G 5 x Q, 77, 77' G Val(A'), 
II 7; — 77' < e and 77, rj' differ only on one clock} 

Note that for aU 7/, 77' G Val(A') said X CX: 

• if ?7 and 77' differ at most on one clock, then so are 77 [X := 0] and 77' [Al := 0]; 
. ||7;[X:=0]-7;'[X:=0]||^ < ||77 - 77'||^. 



15 



Suppose {s,q) e S X Q and ry,?;' G Val{X) which satisfies ||?7 — ri'\\^ < e < 1 
and differ only on the clock x, i.e., r]{x) ^ ri'{x) and r]{y) = r]'{y) for all y ^ x. 
W.l.o.g we can assume that r]{x) < r]'{x). We clarify two cases below. 

Case 1: int(77(a;)) = int(r7'(a;)). Then by r](x) < ri'{x), wc have frac(r7(x)) < 
frac(77'(a;)). Consider the "behaviours" ofrj + t and tj' + t when t goes from to 
oo . We divide [0 , oo ) into open integer intervals (0,1),(1,2),..., (Tmax — 1 , Tmax ) 
and (Tmax. oo)- For each n < r,„ax, we further divide the interval (n, n+ 1) into 
the following open sub-intervals: 

(n, n+l— frac(77'(x))), (n+1— frac(77'(a;)), n+l— frac(77(a;))), (n+1— frac(?7(a;)), n+l) 

One can observe that for i G {n,n + l — ii ac {rj' (x))) U (n + 1 — frac(77(a;)), n + 1), 
we have rj + t =g rj' + t, which implies that r] + t and rj' + t satisfies the same 
set of guards in A. However for t e (n + l — frac(?7'(a;)), n + 1 — frac(r7(a;))), 
it may be the case that 77 + t^gf]' + 1 due to their difference on clock x. Thus 
the total length for t within {n,n+ 1) such that 77 + t^o-r]' + t is smaller than 



\v{x) 



i]'{x)\. Thus we have (f): 



J 



\{s)e 



-\(s)t ^ 



< 



< 



^ P(s, u) ■ h(u, q^+S (7? + t) := 0]) 

ues L 

- P{s, u) ■ h(u, q^'+\ W + t)[X^;+* := 0]) 
/ {A(s)e-^(*)*-5(e)} dt + X{s)e-^^'^'' ■\r]{x)-T]'{x)\ 



i-n+l 

5{e)- / |A(s)e-^(^)*| di + A(s)e- 

J n 



X{s)n . 



Note that for all t e (T^ax, 00) and X CX,{t] + t)[X := 0] =b (V + t)[X := 0]. 
This implies h{u,q^+J,{r, + t)[X.^+J := 0]) = h{u, ci^'+' , {rj' + t)[X.^,'+' := 0]). 
Therefore we have (i): 

\h{s,q,r]) - h{s,q,r]')\ 



< 



n=0 



< Sie) ■ {Xis)e-^^'^*} dt + X{s) ■ € ■ £ 



-X{s)n 



n=0 



< 
< 



(5(e) •(l-6-^W^—) + c.A(5). Tmax 
(5(e) • (1 - e-^-«^'— ) + e • A^ax • T^a, 



Case 2: int(ry(a;)) < int(77'(a;)). By \r]{x) — r]'{x)\ < 1, we have int(?7(a;)) + 1 = 
int(?7'(a;)) and frac(77'(a:;)) < frac(?7(a;)). Similarly we divide the interval [0, cx)) 
into integer intervals (0, 1), (1, 2), ... , (T,„ax - 1, T^ax), (Tmax, 00). And in each 
interval (n, n+l), we divide the interval into the following open sub-intervals: 

{n, n+l— frac(77(x))), (n+l— frac(ry(x)), n+l— frac(ry'(a:))), (n+l— frac(ry'(x)), n+l) 

If i <E (n + l — frac(7y(a::)), n + l — frac(?7'(.T))), then rj + 1 =g rj' + t. And if t lies 
in either (n, n + 1 — frac{r]{x))) or (n+l — frac(77'(a;)), n + 1), then it may be 
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the case that rj + t^gf]' + t. Thus the total length within {n,n + 1) such that 
rj + t^gf]' + t is still smaller than \ri{x) — rj'{x)\. Therefore we can apply the 
analysis (f) and (|), and obtain that 

\his, 9, 77) - Hs, q, ry') I < <5(e) • (1 - e"^— ) + e • A„ 

Thus by the definition of (5(e), we obtain 

d{e) < die) ■ (1 - e-^— ^--) + e • A„,ax • Tn: 



which implies 6(e) < e ■ e-^"--^"-- • A,„ax • Tiiiax ■ By letting e = \\ri - we 
obtain the desired result. □ 

Corollary 2. For all {s,q) € S x Q and ri,r]' € Val(A'), if \\ri - r]'\\^ < 1 then 

\prob{s, q,ri) - prob(s, 9, 77')! < Mi • ||?/- 
where Mi is defined as in Theorem^ 

Proof. Directly from Corollary [ij Lemma |6] and Theorem |2] □ 

By Lipschitz Continuity, we can further prove that prob is the unique solution 
of a revised system of integral equations from the one specified in Corollary [T] 

Theorem 3. The function prob is the unique solution of the following system 
of integral equations on h : V t-^ [0, 1]: 

1. for all s E S, q <E Q and rj^rj' G Val(A'), if r/ =b rj' then h{s,q,r]) — 
h{s,q,ri'); 

2. for all (s, (7, rj) G V, h{s, q,r/) ~ if (s, g, [r]]^) cannot reach a final vertex 
in G, and h{s, q,rf) — 1 if q ^ F; 

3. for all (s, q, rf) G V, if (s, g, [?7]~) can reach a final vertex in G and q ^ F , 
then h(s, q, rj) equals 



+ C30 



A(s)e 



-A(s)t 



J2 P(s, u) ■ h (u, q^+S iv + ore* 0]) 



dt 



Proof. By Corollary [T] Lemma [8] and Lemma [6] prob satisfies the referred 
integral-equation system. Below we prove that the integral-equation system 
has only one solution. 

We first prove that if /i : V [0, 1] satisfies the integral-equation system, 
then h satisfies the prerequisite of Theorem[2j We only need to consider h{s, q, rf) 
such that (s,g, [t?]^) cannot reach a final vertex in G. Note that h{s,q,vi) — 0. 
From the proof of Lemma [8| we can construct a disjoint open interval cluster 
X such that: (i) IJX C R>q and R>o\lJ^ finite; and (ii) for all / G I and 
t,t' £ I, 7] + t ^ ?] + t' . Choose any t e [JX and u £ S such that P(s, u) > 0. 
Then (u,q^+*, [{r] + t)[X.'^+J := 0]]^) cannot reach some final vertex in G since 
[ri + t]^ is not marginal. Thus h{u,q'^+J, {ri + t)[X'^+* := 0]) = 0. It follows that 



A(s)e 



-X(s)t 



Y,^{s,u)-h («, q^+*, + t) re* := 0]) 



Lues 



dt^O . 
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Now suppose /ii,/i2 : V i-> [0,1] are two solutions of the integral-equation 
system. Define h := \hi — /i2|. Then By Theorem[2] h is continuous on Val(A'). 
Further by the fact that /i(s, q, rf) = h{s, q, rj') whenever rj =b 77', the image of h 
can be obtained on 5 x Q x ri^eA'I'^' -^a;]- Thus the maximum value 

M := sup{/i(s, g, 77) | (s, q, 77) G V} 

can be reached. Below we prove by contradiction that M = 0. Suppose M > 0. 
Denote Q := {{s,q,ri) G V | h{s,q,r]) = M} . We first prove that (f): for all 
(s, q,ri) £ Q and all edge (s, q, [77]^) — > (s', q' , r') in G, there exists 77' G r' such 
that {s^q',^) G Q. 

Consider an arbitrary {s,q,ri) G Q. By M > 0, we have (5,5, [77]^) can 
reach a final vertex in G and q ^ F. As before, we can divide [0, +00) into 
a cluster X of open intervals, disregarding only a finite number of isolating 
points t, such that [7/ + r]^ (r G /) does not change for each I E I. Thus 
h (w, qJJ^*, (7/ + t)[X^'i:* 0]) is piecewise continuous on t G IR>o, for all u £ S. 
Note that 



h{s,q,r]) 



















<-r\ 









5]P(s,7.)./7(7.,q^+*,(77 + t)[X^+* :=0]) 



.ues 



dt 



^P(s,u) • h{s,q,ri) 



.ueS 



dt 



h{s,q,vi) 



By the piecewise continuity, h {u, q^^*, (7; + t)[XJ^+* 0]) = M whenever t G 
[jl and P(s,7i) > 0. Note that [0, -|-cx))\ IJI is finite. Thus for all edge 
(s,g, [77]^) {s',q',r') in G, there exists t G [Jl such that q^'"^* = q' and 
(77 + t)[X'J+* := 0] G r'. It follows from (s', g', (r/ + i)[X;'+* := 0])) G Q that (f) 
holds. 

Let (s,q,i]) G Q. Then there exists a path 

(s,g, [7/]^) = {sQ,qo,ro) {si,qi,ri) . . . (s„,g„,r„) 

in G with g„ G -F. However by (f), one can prove through induction that there 
exists 77' G r„ such that isn,qmV') € Q, which implies ^ F. Contradiction. 
Thus M = and h{s, q, 77) = 0. □ 



5 A Differential Characterization 

In this section we present a differential characterization for the function prob. 
We fix a CTMC M = (S, L, P, A, C) and a DTA A ^ (Q, L, X, A, F). Below we 
introduce our notion of derivative, which is a directional derivative as follows. 

Definition 12. Given a function /i : V n- [0, 1], we denote by V^/i and resp. 
V^/i the right directional derivative and resp. the left directional derivative of 
h along the direction 1 if the derivative exists. Formally, we define 

• Vj/i(s, q, rj) :— lim j ■ {h{s, q,ri + t) ~ h{s, q, 77)), if the limit exists. 
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• Vi h{s, q, rf) := lim \ ■ (h(s, q, rj) — h(s, q,rj~t)), if rjix) > for all x E X 
and the limit exists. 

for each (s, q, r/) e V. 

Below we calculate these directional derivatives. 

Theorem 4. For all {s,q,?]) G V with q ^ F, V]^prob(s, g, 77) exists, and 
V^prob(s, g, 77) exists given that ri{x) > for all x G X. Furthermore, we 
have 

Vf prob(s,(j,77) = A(s)-prob(s,g,77)-A(s)-^ P(s,u)-prob(u,q^^^,77[X^'^^ 0]) 



ttes 



and 



Vi prob(s,g,77) = A(s) • prob(s, g, 77) - A(s) • ^ P(s, u) • prob(u, q',' ?7[X^ := 0]) 



whenever Vj^ prob(s, g, 77) exists. 

Proof. We first prove the case for V^prob(s, g, 77). By Corollary [l] 
prob(s,g,7, + i) = r (A(s)e-^(^)-- [^P(s,u)- 

prob(z., q^+(*+-), (77 + (t + r))[X^+(*+-) 0]) 



dr 



for i > 0. Note that the integrand function is Riemann integratable since it is 
piecewise continuous on r. By the variable substitution r' = i + r, we have for 
t > 0, 

prob(s,g,77 + i) = e^^'^'^ ■ T lx{s)e^^^'^^-\^V{s,u)- 



ues 

prob(zi,q;',r,(^ + r)[X;',r := 0]) 

Then we have 

prob(s, q,r] + t)~ prob(s, g, rj) 



dr 



+ 00 



+ 00 



jA(s)e-^(")" 



A(s)e 



+00 



lues 
A(s)e-^(") 



dr 



J2 P(5, • Prob (u, q2+^ + 0]) 1> dr 

ties 

^ P(s, Zi) • prob (7., q;'+^ (Ty + r)[X;'7 := 0]) 

J2 P(s, ^) • prob {u, q^+^ + r)[X^,T 0]) 
^P(s,^i)•prob(^.,q^+^(77 + T)[X2+" :=01) ^ dr 



dr 



Ugs 
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By Lemma Pzl there exists ti > such that q^t^ and X^"^"^ does not change for 
T G (0,ti). Thus the integrand function in the integral 



A(s)e 



-\{s)t . 



J2 P(s, u) ■ prob {u, q^,r, {rj + r)[X^+/ 0]) 



dr 



is continuous on t when t S (0,ii), and the point r = can be continuously 
redefined. Thus by L'Hospital's Rule, we obtain 

V+prob(s, q, rj) = A(s) • prob(s, q, rj) ~ X{s) • J] P(s, u) • prob(«, qf„ 77[xf, := 0]) 

Then we handle the case for Vj^prob(s, q, ry) given that ri{x) > for all x G 
Val(A'). For t e (0, min{r/(a;) | x e X}), we have 

prob(s, q, rj) — prob(s, q.rj ~ t) 
= prob(s, q, [rj-t) +t) - prob(s, q.-q-t) 

+00 C 



+ 00 



A(s)e 



-X{s)i 



1) 



A(s)e 



(1 _ e-^('^)-*) 



-\-oo 



A(s)e 



^ P(., u) ■ prob q(7*)+^ ((,7 - <) + r) [X(V*)+- := 
ties 

^ P(s, I.) . prob q(7*)+^ ((77 - t) + r)[X(7*)+- := 0]) 
^ P(s, • prob U q,^,t^^"*^ + (r - t))^^^ 



:u£S 



-A(s 



-A(s)r . 



+ 00 



^ P(s,u) • prob (u, q;';(*--), (ry - (< - r))[X2.-(*-^) := 0] 



-A(s)-t 



A(s)e 



-A(s)r . 



lu£S 



A(s)e 



A(s)r 



^ P(s, li) . prob (7., q^_-^ (ry - r) [X^'-^ 0]) 



^ P(s, u) ■ prob (^., q^+^ (77 + r)[X^+- 0]) 



dr 



where the last step is obtained by performing t' = r — t in the first integral and 
t' = i — T in the second integral. By Lemma [?[ there exists ^2 > such that 
q^~* and X^~* does not change for t G (0,t2)- Thus the integrand function in 
the integral 



A(s)e 



A(s 



^ P(s, u) • prob {u, q^_-^ (,7 - r)[X^;^ := 0]) 
L«es 



dr 



is continuous on r when t € (0,i2)- Furthermore, the point r = can be 
continuously redefined. Thus we can also apply L'Hospital's Rule and obtain 
the desired result. □ 

Remark. Note that if [77]^ is not marginal, then [77^]^ = [77 ]~ = [77],^. This tells 
us that Viprob(s, g, 7;) exists when [7;]^ is not marginal, i.e., \7^prob{s,q,ri) — 
Vi prob(s,g,77). □ 
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Based on Theorem [4] we present our differential characterization. 

Theorem 5. The function prob is the unique solution of the following system 
of differential equations on h :V ^ [Q, 1]; given any s Cz S,q Cz Q and rj,?]' e 
Val(A'), 

1- ifv =b v' then h{s,q,r]) = h{s,q,r]'); 

2. h{s,q,r])—Oif{s,q,[ri]^) cannot reach a final vertex in G , and h{s, q,r]) — 
1 ^fq&F; 

3. if (s, q, [rj\^) can reach a final vertex in G and q ^ F, then 

y+h{s, q, ry) - \(s) ■ h{s, q, ry) - \{s) ■ ^ P{s, u) ■ h{u, qf„ 77[xf, 0]) 

ues 

and 

\7^his,q,v) = \{s) ■ his,q,v) - A(s) • ^ P(s,u) • h{u,ci^l^,r^[Xl^ 0]) 

when r]{x) > for all x ^ X . 

Proof. It is clear that prob satisfies the differential equations above. For the 
uniqueness, we prove that all functions h that satisfies the differential equations 
will satisfy the integral equations specified in Theorem [3j Let h be such a 
function. The situation is clear when (s, g, [rj\^) cannot reach a final vertex in 
G 01 q G F. Below we only consider {s,q,rj) such that {s,q, [r]]) can reach a 
final vertex in G however q ^ F. For each such (5,(7,77), we define f[s,q,ri] : 
M>o [0,1] by f[s,q,r]]{t) :— h{s,q,ri + t). Then /[s,g, 77] is differential at 
those points t where [rj + t]^ is not a marginal region. Note that there are only 
finitely many points t such that [rj + t]^ is marginal, we can divide [0, +(xi) 
into a finite cluster I of open intervals, disregarding a finite number of isolating 
points, where for each / el, rj + t ^ i] + t' for allt', t € /. Thus f[s,q, rj] is 
piecewise differentiable on ]R>o. Then f[s, q, 77] is Lipschitz continuous since the 
existence and boundness of V^h and Vj^/i. 

Consider t G ]R>o such that [7; + i]^ is not marginal. We have 

^^f[s,q,rm^ (2) 
\{s) ■ h{s, q,v + t)- A(s) • J2 P(«. • ^(«' iv + := 0]) 

Multiply each side of ^ by e"'*'*^"^*, we obtain that the equality 
e-^(^^* • ^/[s, q, vKt) - A(s)e-^(^)* • f[s, q, Tj]{t) = 

This is essentially 

- A(5)e"^(^)* . J2 u) ■ h{u, q^+S iv + t) [X^.t* := 0]) 
lies 
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Note that e"'*'^'*^* • f[s,q,'q]{t) is absolutely continuous on any closed interval 
since it is Lipschitz continuous. Thus by the Fundamental Theorem of Calculus 
for Lebesgue Integral [19] , for each / G I with / = (inf /, sup /) , 



sup / 
inf 7 



Then by 



we have 



r+oc 

•^0 ues 



lex 



sup I 



inf 7 



-h(s,q,ri) 



Thus we obtain 



+ 00 



A(s)e 



-A(s)t 



^ P(s, • q;'+\ (77 + t)[X;'+* 0]) dt 



ties 



Then h satisfies the prerequisite of Theorem [3] So h is unique. 



□ 



6 Finite Difference Methods 

In this section, we deal with the approximation of the function prob through 
finite approximation schemes. We establish our approximation scheme based 
on Theorem [s] and by finite difference methods [5D] . Then we prove that our 
approximation scheme converges to prob with a derived error bound. 

We fix a CTMC M = {S,L,T',\,C) and a DTA A = iQ,L,X,A,F). For 
computational purpose we assume that all numerical values in A4 are rational. 

Given valuation j] and t > 0, we define 77 © i G Val(A') by: (77 © t){x) — 
min{ra;, 77(2;) + 1} for all x e X. Note that 77 © = 77 iff 7?(x) < for all clocks 
X. We extend ©, +, A(.), [.]^ and P(.,-) to a triple (5,9,77) € V as follows: 

• (s, q,r])®t^ {s, q,ri®t) and (s, q,'q)+t= (s, 9, 77 + t); 

• [(s,g,77)]^ = {s,q, [77]^); 

• A(s, q, T]) — A(s) and P ((s, q, ri),u) — P(s, u) for u G S. 

Furthermore, we say that [{s,q,ri)]^ is marginal if [77]^ is marginal. Note that 
by Lipschitz Continuity and Lcmma[6j we have prob(v + t) = prob(u © t) for all 
w G V and < > 0. 

Given (s,q,r]) G V and 7i G 5, we denote (s,q, 7/)+ := (s, g,7;+), and denote 
the triple (li, qf„ 77[Xf, := 0]) G V by (5,9,77)+. 

Before we introduce our approximation scheme, we first prove a useful lemma. 
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Lemma 9. Let w S V such that [w]^ cannot reach a final vertex in G. Then 
[v t]^ cannot reach a final vertex in G for all t > 0, and [v^]^ cannot reach 
a final vertex in G for all u € S such that P{v,u) > 0. 

Proof. Since [v]^ cannot reach a final vertex in G, we have [u + t]^ cannot reach 
a final vertex in G for all < > 0. Note that v + t and v Q) t differ only at those 
clocks X whose values in v + t are greater than T^. Then [v + t]^ can reach 
a final vertex in G iff [v Q) t]r^ can reach a final vertex in G by the fact that 
{v + t) + T ^ (v Q) t) + T ioi all T > 0. Thus we have [v (B t]r^ cannot reach a 
final vertex in G. 

Let u & S such that P(w, u) > 0. By Lemma[7] there exists ti > such that 
[u + i]^ does not change for all t e (0,ti). Then [w + i]^ is not marginal for 
t e (0, ti). Since [v]^ cannot reach a final vertex in G, we have cannot 
reach a final vertex in G for t e (0,fi). Then by Lemnia[8j prob((i; + = 
for all t G {0,ti). Thus by Corollary [2] we have prob(i;+) = 0. It follows that 
[^^]~ cannot reach a final vertex in G. □ 

6.1 Approximation Schemes 

We establish our approximation scheme in two steps: firstly, we discretize the 
hypercube HsGA'P'^a:] — Val(A') into small grids; secondly, we establish our 
approximation scheme by building constraints between these discrete values 
through finite difference methods. By Lipschitz Continuity and Lemma |6j we 
don't need to consider clock valuations outside IlxGA'P'-^a:]- '^^^'^ discretization 
is as follows. 

Definition 13. Let m S N. A clock valuation ry is on m-grid if 77(2;) e [0,Tj;] 
and ri(x) • m is an integer for all clocks x. The set of discrete values Dm of 
concern is defined as follows: 

Dm — {h[(s, g, 77)] I {s,q,r]) £ V and 77 is on ?Ti-grid} . 

Below we fix a ?7i G N and define p :— m^^. Based on Theorem [5j we 
establish our basic approximation scheme as follows. 

Definition 14 (Basic Approximation Scheme). The approximation scheme Fm 
consists of the discrete values and a system of linear equations on Dm- 
The system of linear equations contains one of the following equations for each 
h[v] e Dm- 

• h[w] = if [v]r^ (as a vertex of G) cannot reach a final vertex in G; 

• h[w] = 1 if [v]^ is a final vertex in G; 

• If [v]^ can reach a final vertex in G however itself is not a final vertex, 
then 

= X{v) ■ h[v] ~ X{v) - 2^ P{v, u) - h[i;+] . 

^ ues 

In other words, we relate elements of Dm by using VjJ^h in Theorem [sj Note 
that h[v] is in essence v. Sometimes we will not distinguish between h[v] and v. 

We note that Tm does not have initial values from which we can approximate 
prob incrementally. One fundamental problem is whether Tm has a solution. 
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or even a unique solution. Another fundamental problem is the error bound 
max{|ft,*[v] — prob(?;)| | h[v] € D„i} provided that h* is the unique solution of 
r 

^ m ■ 

Below we first derive the error bound of which is the error bound of each 
linear equality when we substitute all h[v] by prob(w). Note that generally the 
error bound of an approximation scheme does not imply any information of the 
error bound of the solution to the approximation scheme. 



Lemma 10. For all h 

final vertex in G then 



v] G D„i, if [v]^ is not a final vertex and can reach some 



- ■ (prob(t; (Bp) — prob(v)) — VjJ^prob(w) 



< M2 ■ p, where 



Proof. Suppose h[v] e D,„ such that [v]^ is not a final vertex and can reach 
some final vertex in G. Since h[v] e the function f[v]{t) : t 1— )■ prob(w © t) 
is continuous on [0, p] and is differentiable on (0, p). By Lagrange's Mean Value 
Theorem, there exists p' G (0,p) such that ^•(prob(w©p) — prob(u)) = ■^f[v]{p'). 
By Theorem [4j we have 

^/H(P') = A(«) • prob(^; + p') - \{v) ■ ^ P{v, u) ■ prob((,; + p')t) 

ues 



and 



Vj"prob(u) = X{v) ■ prob(w) — A(w) • ^ P{v,u) ■ prob(i;„ ) 

ues 



Then by Corollary [2j we obtain the desired result. □ 

To analyze we further define several auxiliary approximation schemes. 
Below we define Bm, B™'''^ as follows: 

• Bm = {^[v] G Dm I [v]^ is not final and can reach some final vertex in G} 

• B;;^'^'^ = {h[v]eBm\v= (s, g, ry) and f]{x) = for aU x e X} . 

For each h[v] G B„i, we denote by Ny G No the minimum number such that 
either h[v ® {Ny ■ p)] G B™^^ or [v ® {Ny ■ p)]^ cannot reach some final vertex 
in G. We first transform r,„ into an equivalent form. 

Definition 15. The approximation scheme F^ consists of the discrete values 
Dm, and the system of linear equations which contains one of the following linear 
equalities for each h[v] G D^: 

• h[w] = if [v]^ cannot reach a final vertex in G; 

• h[w] = 1 if [v]r^ is a final vertex of G. 

• If h[v] G B„AB^f'', then 

• ifh[f](SB»"t.licnh[«| = E,.MP(».»)-hKll ^ 

It is clear that F^ is just a re- formulation of F„i. Note that the case h[v] G 
B'^'^^ is derived from v ® p — v. The error bound of FJ^ is as follows. 
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Lemma 11. For all U[v] G B'^'''', prob(w) ^ E„esP("'") ' Pi'ob(w+). For all 

h[v] e B„ABr^ 

Proo/. The case h[v] e B™'''' is due to the fact that prob(w) = 0. The case 



h[v] e Bm\B™^'^ can be directly derived from the statement of Lemma 10 □ 



Remark. rj„ can somewhat be viewed as a reachabihty problem on a discrete- 
time Markov chain (DTMC). The nodes of the DTMC are elements of D^. 
The goal nodes are those h[w]'s where [w]^ is a final vertex of G. The set of 
nodes which can not reach a goal state are those h['y]'s where [v]^ cannot reach 
a final vertex in G. Then states the relationship between the reachability 
probabilities of the remaining states. It seems that we can use this fact to 
deduce that (hence F^) has a unique solution (e.g., by applying the proof 
on [5j Page 766]). However, this may fail because the remaining states may still 
contain states which cannot reach goal states. This can be seen as follows. 

Suppose the DTA has four locations {go, 9i) 92, 93} where 52 is the only fi- 
nal location and two clocks {x,y}. Let s be a CTMC-state with P(s,s) > 0. 
Assume that the DTA has only two meaningful rules, namely {qo,C{s),x < 
1 Ay < l,{a;},gi) and {qi,C{s),x < 1 Ay > 1,0,(72)- The other rules lead 
to the "deadlock" location q^. Define 77 = (^^^^— ^,0) where the first (resp. 
second) coordinate is the value on x (resp. on y). Then {s,qo,[r]]^) can 

reach a final vertex in G since we have {qo,ri) — ^ ^ ^> ((71, (0,^ • p)) and 

(?!, (0, 1 • p)) — ^-A {q2, (1 — I • p, 1 + I • p)). However after the discretiza- 
tion, the node (s, go, v) can go to (s, qi, (0, 0)) but from (s, gi, (0, 0)) we cannot 
go to 52- This implies that {s,qo,ri) cannot reach a final node. □ 

Below we unfold F^ into another equivalent form FJ^. 

Definition 16. The approximation scheme F^ consists of the discrete values 
Dm, and one of the following linear equality for each h[i;] e D.,„: 

• h[?;] = if [v]^ cannot reach some final vertex in G, and h[w] = 1 if [v]^ is 
a final vertex; 

• if h[v] e B„AB;^,^'', then 




(4) 

where f{v) := if [u © [N^ ■ p)]^ cannot reach some final vertex in G and 
f{v) ■■= Eu^s P(«. • h[(« ® {N. ■ P))+] if h b © {N, ■ p)] G B--; 
. if hH G B-- then hH = Eues^i^^^) ' ^K] ■ 

Intuitively, F"j is obtained by unfolding h [u © p] further in Equation ([S]) . In 
the following, we prove that F'^^ and F^ are equivalent. 
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We describe by a matrix equation = A/i + b where fi is the vector over 
Bm to be solved, b : i— >■ K is a vector and A : B^ x B^ H' K is a matrix. 
More specifically, the row A(h[w], — ) is specified by the coefficents on h[v'] £ Bm 
in Equation Q; the value b(h[t;]) is the sum of the values over Dm\B„i in 
Equation Q. The exact permutation among B„i is irrelevant. Analogously, we 
describe by a matrix equation /i = C/i + d. 

Lemma 12. F'^^ andV^ are equivalent, i.e., they have the same set of solutions. 

Proof. "FJ^ =4> F^": It is clear that F^ is obtained directly from F^ by ex- 
panding h[v © p] iteratively in Equation ^ whenever v ® p £ Bm\B™'^'^. 

'T™ ^ F'^": Let {h[v] \ h[v] € D„} be a solution of F'^. We define 
h : D,n M by h[v] — h[v] ■ {Bm){v) for h[ti] e We prove that for all 

h[v] e B„\Bf^^'^ and all < m < iV„, 

-£|(tt;^)'-'<M"»('-.)1)| 



^ \ m+1 



l + p-X{v) 



h[v®{{m + l)-p)] . (5) 



We prove this by induction on Ny — m. The case when m — Ny — 1 is directly 
specified by r"^. Let 0<m<m + l<Ny and suppose Equation [s] holds when 
Ny' — m' < Ny — m. Then wc have 



\ m+2 



+ A(i;) 
Then we have (f): 



h[v®{{m + 2) ■ p)] 



M.i^|{(rT^)'r|^^|:p(".«)-'.[(-'-< 

■m+l 



+ d{h[v®{{m+l)-p)])+(^^^-^-^^ ■h[v®{{m + 2)-p)]} . 
Note that iV^^^^+i^.p — = A^t, — (m + 1) < Ny — m. Thus by the induction 
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hypothesis, 

h[v(Biim + l) ■ p) 
p-X{v) 



l + p-\{v) 



Y,nv,n)-h[{v(B{{m + l)-p))l] 



d(h[i;®((m+l)-p)]) 



l + p-\{v) 



h[v®i{m + 2)- p)] 



Thus we obtain that Equation ([s]) holds when we substitute h[v(B{{m+l)-p)] into 
(f). Then by taking m = in Equation ([5|, we obtain that {h[v] \ h[v] G D„i} 
is a solution of F^. □ 

Below we derive the error bound of T'^. 

Lemma 13. The error bound ofV^ is M3 • p, where M3 — T^ax • -^2- 

Proof. We only need to consider h[t;] e B„j\B™^''. By Lemma 11 

p-\{v) 



prob(u) 



1 



l + p-\{v) 



prob(w © p) 



l + p-\{v) 



^V{v,u) ■ prob(i>+) 



ties' 



< M2P^ 



(6) 

Expanding prob[w p] one step further in ([6]) will result in another error of 
i+p.\(^y) ' M2p'^. By iterated expansion up to (< TmaxP^^) steps, we obtain 
that the error bound of r"^ is no greater than M2p^ ■ X^^^o ^( i+p\(s) which 
is smaller than M2 ■ Tmax ■ P- D 



6.2 Analysis of the Approximation Schemes 

Below we analyse the approximation schemes proposed in the previous subsec- 
tion. We fix some m E N and p = rn^^. We define A,nin = min{A(s) | s e S*} 
and Pinin = min{P(s, u) \ s,u G S and P(s, u) > 0}. Note that Amin > 0. 

Recall that we describe by /x = + b and by ^ = Cfj, + d in 
the previous subsection. Below we analyse the equation fi = Ap + b. To this 
end, we first reproduce (on CTMC and DTA) the notions of S-seperateness 
and S-wideness, which is originally discovered by Brazdil et al. on semi- 
Markov processes and DTA. Below we define the transition relation over V 
by: (s, q, -q) 4 {u, q' , 77') iff P(s, u) > and {q, rj) iil^ (^'^ r]'). 

Definition 17. A clock valuation rj is 5-separated if for all (ii,rf2 G T?.?;, either 
di — d2 or \di — 6^2! > ^. A transition (s, q, if) (u, q' , rj') is S-wide ii t > S and 
for all T e {t — S,t + S), rj + T r] + t. Furthermore, a transition path 

(so,90,'7o) ^ {si,qi,Vl) ■■■ ^ iSn,qn,Vn) > 1) 

is S-wide if all its transitions are 5-wide. 

The following result is the counterpart of the one on semi Markov processes 
and DTA [TOl [11] . Below we say that a set X of disjoint open intervals is an 
open partition (of [0, 1]) iff it holds that IJI C [0, 1] and [0, 1]\1JI is a finite 
set. Given a non-empty open interval / C [0, 1] with / = (ci, C2) and a t G R>o, 
we denote by / o t the (possibly empty) interval (frac(ci + t), frac(c2 + t)) . 
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Lemma 14. For all {s,q,rj) € V, if rj is S-separated and (s, [77]^) can reach 
some final vertex in G however q ^ F, then there exists an at most \V\-long, 
d/\V\-wide transition path from {s,q,ri) to some {s',q',r]') with q' £ F. 

Proof. Let (s, q,r]) € V such that (s, q, [r]]^) is not final and can reach some final 
vertex in G. Then there is a path 

{s,q, [vU) = > {Sn,Qn,rn) 

in G such that 1 < n < jV^j and qn & F. We first inductively construct a 
transition path 

Vn) 

such that r]i G ri for all 1 < i < n, while maintaining the following structures: 

• two open partitions with TZr^ C [0, 1]\ IJX^ for each 1 < i < n; 

• a bijection ipi : I[ 1— >■ Ij for each 1 < i < n; 

• two intervals {c\,C2) e Ij, (w|,w|) e I- such that (pi((w|,w^)) = (01,02) 

for each 1 < i < n — 1; 

• a c* e (c^^, C2) for each \ < i < n — 1. 

Initially, we set rji = rj and Ii = X[ = {{wj,Wj^i) | < j < m}, where 
{'Wj}o<j<m satisfies that TZr, = {wo, wi,. . . , Wm} and wj < Wj+i for all < j < 
m (note that w;o = and w„i = !)• We let ipi be the identity mapping. 

Suppose that the transition path until (sj, qi, rji) together with ipi are 

constructed. Since {si,qi,ri) (sj+i, gi+i, rj+i), there exists ti > such that 
[Vi+ti]'^ is not marginal, q^+i = q^;.t*' and (77^ + t,)[X;^''+*' := 0] G r^+i. Since 
[Vi+til^ is not marginal, we can further assume 1 G (w^ +frac(t8), +frac(fi)) 
for some (w^,w|) G X^'. Denote (c^,c|) = (fi{{w\,\Nl)) and choose c' G (c\,C2) 
arbitrarily (e.g., c* = i • {c\ + cD). Then we set rji+i := (77^ + ti)[X^*+*' := 0] G 
fj+i, and split as follows: 

li+i :=(^.(Tl-{(wl,w^)})U{(cl,e),(e,e2)} ; 

:={(wi + frac(tO, 1), (0,frac(w^ + frac(ii)))} U {Joti | J G - {(wl, w|)}} 

The mapping v'j+i : I'i+i 1-^ is defined as follows: 

(^i+i((w'i +frac(ti),l)) = (ci,e) and </9i+i((0,frac(w^ + frac(ti))) = (c',4) ; 
iPi+i{Ioti) = ipiil) for all {(wi, w^)} . 

Intuitively, we record by every splitting point which may make r]i + ti less 
wide, and we record the splitting information without time delaying by Ij, where 
the correspondence between them is maintained by tpi. 

Since n < at most |y| — 1 splitting occurs at each interval (including 
its sub-intervals) in Ii during the inductive construction above. Based on this 
point, we inductively construct a 5/|F|-wide transition path 

{s,q,'n) = (si,gi,??i) ^ ... (sn,?™,??^) 

such that ry- ~ 77^ for all 1 < i < n, while maintaining an open partition I" and 
a bijection ipi : I" ^ I- for each 1 <i <n which satisfy the following condition 
for all 1 < z < n: 



28 



1. 7^,- c [o,i]\u^r; 

2. for all /i,/2 e Z-', sup/i < inf /2 iff supV'*(/i) < inf V'i(/2); 

3. for all X e X, (c'^jCj) G I" and (ci,C2) S if V'i((c'i, 4)) = (ci,C2), then 
(i) r7-(a::) = c[ iff 77^(0;) = ci and (ii) ri[{x) = iff rnix) = ci. 

Intuitively, we maintain the order on the fractional values in the previous tran- 
sition path, however adjust the gap on each transition. 

Initially we set I" = and i/'i to be the identity mapping. Suppose the 
path until {si,qi,r]'^) is constructed. Let {d\,d2) G If such that ?/^i((d|,d|)) = 
(w*i, w5,). We choose t'^ such that int(t9 = int(t,), 1 G (d^^ +frac(i9, d| +frac(t^)) 
and the length of (d^^ + frac(i:^),l) (resp. (0, frac(d2 + frac(t^))) is no smaller 
than {k\ + 1) • S/\V\ (resp. (/c2 + 1) • (5/|y|), where k\ (resp. ^2) the number 
of splittings on the sub-intervals of (c5^,c*) (resp. (c*,c|)) during the previous 
inductive construction. Then we set yy-^j^ :~ {ri[ +i'J[Xgi^*' := 0] and similarly 
split ,ipi as follows: 

X'Ui := {/ o / e I': - U {(dl + frac(t:), 1), (0, frac(d^ + frac(tD))}; 

ti)^+i{I' ot[) =IoU whenever = I and /' ^ (di,d^); 

i>^+l{{d\ + frac(tO, 1)) = (wi + frac(<,), 1); 
i/'»+i((0,frac(d*2 + frac(iD))) = (0, frac(w^ + frac(<,:))) . 

By the choice of t'^ , ti , one can prove inductively that r]i ^ t][ and r]i +ti ^ ri[ + 1[ 
for all 1 < i < n. Thus the constructed path is a legal transition path. And by 
the construction, this transition path is 5/|y|-wide. □ 

Then we study the linear equation system /i = A/x -I- ^ where : 1— ^ M is 
an arbitrary real vector. Below we define <^max : '-^ K such that Cmax(h[w]) = 
Mg-pforaU h[v] G B„,. We denote by |C| the vector such that |C|{hH) = |C(hM)| 
for all h[v] G B,„. We extend < to vectors over B„i in a pointwise fashion. 

Lemma 15. Suppose m > 2\V\'^. Let C, be a vector over B^ such that 
ICI ^ Cmax- Then the matrix series X^^o-^"^ converges. Moreover, we have 
II Er=o A"C l|oo< 1^1 • c-l^l • (M3 • p), where c := e-^-^'-^-'' .p„,i„ • ^^fe" ■ 

Proof. Let 6 := |y|-2 and k lm/\V\^\. We analyse (E^^o A"C)(h[w*]) for 
each h[t;*] G B„j. Firstly, we consider the case when ( = Cmax and h[i;*] G B™'^^ 
Denote v* — {s* ,q* ,ri*). By definition, 77* is 1-separated. Then by Lemma 14 
there exists a shortest |y|^^-widc path 

{s*,q*,T]*) = (Si,(ji,77i) ^ ... {Sn,qn,Vn) 

with 1 < n < |y|, ^ F for 1 < i < n - 1 and g„ G F. Note that [rji + ti\^ 
is not marginal for 1 < i < n — 1. We adjust the delay-times in the transition 
path up to (5 by: 

(S ,q ,V ) ^ (Sl,qi,Vl) > ■ ■ ■ 5- {Sn,qn,Vn) 

where Si G [0,6) for all 1 < i < n — 1. Given arbitrary {Si}i, one can prove 
inductively that (f): 

Qi = 9ij v'i =g Vi and ry- + (t^ + Si) =g rj^ -|- for all 1 < i < n - 1 , 
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by checking that (t): r]l{x) + {U + Si) < Vi{x)+U+J2^^^Sj < tj^{x) + ti + \V\-^ 
for all clocks x at each stage of induction. 

Define {Vi}i<i<„ with each C V as follows: 

Vi = {v*} and V,+i = {{v + t)1^^ | u G Vi,r e [U,U+6)} . 

-By (t)j we have [v]^ is not final and can reach some final vertex in G for all 
V e Ur=i^ Vj. Let := {w © I w e VJ for each 1 < « < n. Note that v (B 
and u only differ at clocks x whose value is greater than T^. Thus we have [v]^ 
can reach some final vertex in G iff [u ® 0]^ can reach some final vertex in G, 
for any w e V. It follows that [v]^ is not final vertex and can reach some final 
vertex in G for all v £ ljr=i ^i- 

Then we define {B,^}i<i<„ with each C Dm by: 

B; = {h[v*]} and B^+i = U{Post.(t.) | e B^, 

where for each v E V: 

. Post,(z;) := {h[{v')+J \ v' G Delay,(t;)} ; 

• Delayi(w) :={v®t\t e [U,t, + S), h[v ® r] e D„} . 

Consider any w G and re [ti,ti + S) with 1 < i < n — 1. By (f), we have 
[v + t]^ is not marginal. Then [{v + t)+]^ = [{v ® It follows that 

{v + © = (u © ''')ti-^i ■ Then one can prove inductively that BJ C for 

all 1 < i < n. It follows that BJ C B„. 

We prove by induction on i > I that for all v G BJ^_j, |(AXmax)('i')| < 
(I - • Msp. Note that A^max < Cmax and ACi < AC2 for all < Ci < (2- It 
follows that A-' Cmax < A*Cmax for all < I < j. 

Base Step: i = 1. Consider an arbitrary v E Q'n-i- By B^^ C V^, we know 
that [v']r^ is final in G for all v' G Post„_i(i;). If u © {Ny ■ p) G Delay„_^(w), 
then h[w © {N^ ■ p)] G ^'^'^ by Lemma |9] Therefore from we have 



1- E A(hH,hK]) > p„ 



h[-u'leB„ 



1 



l + p-\{v) 
I 

l + p-\{v) 



^ Prain 

> c 

Otherwise, from Lemma |9] there are at least [S/p\ — k distinct elements in 
Delay„_i(w). Note that kp > ^\V\~'^. We have 

I- E A(hH,hK]) 

hlv']eB,„ 

1 \^."-/p fcp.A(i;) 



> 



> e 



p-Xiv)J l + p-X{v) 



l + i|y|-2.A(«) 

Amin 



mm 



2\V\^ + X„ 
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Thus WC have |(ACmax)(f)| < (1 - C) • M3/J. 

Inductive Step: Suppose (A*Cmax)(w) < (l-c*)-M3p for all v G B;^_,. Wc 
prove the case for i + Fix some t; e B^_^.^^^. liv(B{Ny-p) e Delay„_(j_|_i)(w), 
then similar to the base step we have 

(A^+'Cmax)(^^) < - Pmin ' ^ p. X{v) ) " " '^^'^ ' ^^"^ 

Otherwise, there are at least [5/p\ = k distinct elements in Delay„_(j_|_i)(w). 
By Post„_(j+i)(w) C B^_. and A'Cmax < Cmax, we obtain that 

(A'+lCmax)(^;) 



'-'min 



Then the result follows. Then we obtain that 

(Al^|-lCmax)(i') < (A'Cmax)O^) < (1 - C*) ' A'hp < (1 - cl^l-^) • M^p 

for all 1 < i < n - 1 and i; e B^.^. Thus A^^^'^Ua^iv*) < (1 - cl^l-^) • Msp 
for all V* G B™*^. 

Now consider an arbitrary v G while ( = Cmax- If cither v(B{Nyp) ^ B™*^^ 
or h[{v ® {Nyp)):^] ^ Bm for some u G S such that P{v,u) > 0, then we have 



(Al^lCmax)(^^) < (AC„,ax)(^^) 



< (1 - e-^— • pmin) • Map 

< (l-cl^l)-M3p 



Otherwise, by 



^P(^;*,tx) • (Al^|-lCmax)((^^*)J) = (Al^lC„.ax)K) < (Al^|-lCmax)(t^*) 

where v* = v (B {N^p), we have 

(A-Al^l-^CmaxK^;) < {i-(--\r-X^^^" -^^""^A-MsP 



l + p\{v) 
< (l-cl^l)-M3p 
Then we have Al^'Cmax < (1 - d^l) • Cmax- It follows that 

A^ 'Cmax ^ (1 Cmax 

for all I G N. Thus by the monotonicity of A, we have X^J^g^'Cmax converges 
since J2ilo A'Cmax is bounded by \V\ ■ c"l^l • Cmax- 
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Finally we consider any ( such that \(\ < Cmax- Note that |A'C| < A*Cmax 
since all entries of A* are non-negative. Thus by Cauchy Criterion, it follows 
that J2Zo A'C converges and || J^Zo \\oo< \V\ ■ c'l^l • M^p. □ 

By Lemma [TSl the system of linear equation /i = Afi + ( has a solution for all 
I CI ^ Cmax when TO > 2|yp. Below we assume that m > 2|yp. The following 
lemmas show that the linear equation has a unique solution. 

Lemma 16. For all solutions fi of fi = Kfi + C with \\ C, \\^ < M^p, we have 
\p.\ < p.*, where p* := I]j^o^'Cmax- 

Proof. Let p be an arbitrary solution of p = Ap + Define p' ~ p* — p. 
By the fact that p* ~ Ap* + Cmax, we have /i'(h[u]) > {Ap'){h[v]) for all 
h[v] G Bm- Suppose there is some h[v] G Bm such that /i'(h[t;]) < 0. W.l.o.g 
we assume that /i'(h[w]) is the least element of {/x'(h[u']) | h[i;'] G B„j}. Denote 
c - Eh[.']eB,„ A(hH,hK]) e [0,1]. We have p'{h[v]) > c- p'{h[v]), which 

implies c > 1. Contradiction. Thus p' > 0. Similar arguments holds if we 
define p' = p* + p. Thus we have \p\ < p* ■ □ 

Lemma 17. The system of linear equations p = Ap + C has a unique solution 
for all C such that ||Clloo ^ M^p. It follows that I — A is invertible where I is 
the identity matrix. 



Proof. By LemmafTS} the system p = Ap + ( has a solution. And by Lemma 16 
all solutions of p — Ap + C are bounded by p* . Suppose it has two distinct 
solutions. Then the homogeneous system of linear equations p ~ Ap has a 
non-trivial solution, which implies that the solutions of /i = Ap + C cannot be 
bounded. Contradiction. Thus p — Ap + C has a unique solution and I — A is 
invertible. □ 

Now we analyse (Fm). In the following theorem which is the main result 
of the paper, we prove that the equation p = Cp + d has a unique solution (i.e. 
I — C is invertible), and give the error bound between the unique solution and 
the function prob. 

Theorem 6. The matrix equation p ^ Cp + d (for F^ j has a unique solution 
p. Moreover, maxf,[^]gB„ lA^(hH) — prob(w)| < \V\ ■ c^'^' • M^p. 

Proof. We first prove that p — Cp + d has a unique solution. Let p = Cp + ( 
be a matrix equation such that || C |loo M2p^. From the proof of Lemma 
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we can equivalently expand this equation into some equation p = Ap + C' with 
II C lloo < ^max/p • M2P^ — M3 • p. Siucc p = Ap + (' has a unique solution, 
we have p — Cp -\- ^ also has a unique solution. Thus I — C is invertible and 
p = Cp + d has a unique solution. 

Now we prove the error bound between p and prob. Define the vector p' 



such that /i'(h[w]) = /l(h[w]) — prob(i') for all h[v] e B,„. By Lemma 11 p' is 
the unique solution of p = Cp + C for some || C ||oo< M2p^. Then p' is also 
the uni que solution of the equation p = Ap + C' for some || C' ||oo< M^p. By 
Lemma[l5j || p' ||oo< \V\ ■ c"!^! • Ahp. □ 

By Theorem [g] and the Lipschitz Continuity (Corollary [2]), we can approxi- 
mate prob(s, q, if) as follows: given e € (0, 1), we choose to sufficiently large and 
some h[u] e such that |prob(u) — prob(s, q,rii)\ < and Afalyjc"!^' ■ p < \e. 
Then we solve the system F^ to obtain /l(h[w]). 
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7 Conclusion and Future Work 



We have shown an algorithm to approximate the acceptance probabihties of 
CTMC-paths by a multi-clock DTA under finite acceptance condition. Unlike 
the result by Barbot et al. [6], we are able to derive an approximation error. 
Chen et al. [13] demonstrated that computing the acceptance probability of 
CTMC-paths by a multi-clock DTA under Muller acceptance condition can be 
reduced to the one under finite acceptance condition. Thus our result can also 
be applied to Muller acceptance conditions. One future direction is to refine 
our approximation algorithm by importing zone-based techniques f^. Another 
future direction is to extend this result to continuous-time Markov decision 
processes (CTMDP) [5| or continuous-time Markov games (CTMG) \TU[ [TB]. 
A more challenging task would be to consider the acceptance probabilities of 
CTMC-paths by a non-deterministic timed automaton. 
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